There are several different vehicular situations that are lumped together and called the distracted or inattentive driver problem. These situations include cases where the driver is wide-awake but eating, putting on make-up, texting and talking on the phone; cases where the driver is physically fatigued and may or may not also be sleepy; medical condition cases where the driver suffers a medical condition such as a stroke or heart attack; cases where the driver is under the influence of alcohol or drugs; and cases where the driver is just sleepy. At least one embodiment of the present invention is primarily concerned with detecting a drowsy or sleepy driver but a number of the other causes of the distracted driver also affect the parameters that are the focus of embodiments of the invention and thus will also be addressed, and are encompassed within the scope of the invention in its entirety. A primary focus of at least one embodiment of the present invention is to measure the heartbeat rate and respiration rate of the driver using electric fields or optics, and through analysis of these rates, predict whether the driver is losing his/her ability to safely operate the vehicle. Driver inattention is a larger problem than driving sleepiness and fatigue.
One study found that fatigue plays a significant role in around 20% of all fatal crashes. Another reported that about 58% of one hundred-seven single-vehicle roadway departure crashes were fatigue-related.
Fatigue has been defined as “a state of reduced physical or mental alertness which impairs performance” (Williamson, A., Feyer, A. and Friswell, R. (1996). The impact of work practices on fatigue in long distance truck drivers. Accident Analysis and Prevention, 28(6), 709-719). Dingus, T., McGehee, D., Hulse, M., Jahns, S., Manakkal, N., Mollenhauer, M., and Fleischman, R. (June, 1995). TravTek evaluation task C3 camera car study. Pub. No. FHWA-RD-94-076. Washington, D.C.: U.S. Department of Transportation, Federal Highway Administration.) states that fatigue is “a neurobiological process directly related to the circadian pacemaker in the brain and to the biological sleep need of the individual” and cannot be “prevented by any known characteristics of personality, intelligence, education, training, skill, compensation, motivation, physical size, strength, attractiveness or professionalism”. Note that fatigue and drowsiness are not the same thing. Some of the indicators of drowsiness include a person having their eyes closed for long time, sudden head movements, touching the face with a hand, strong blinking and yawning.
In the medical area, drivers suffering from obstructive sleep apnea syndrome (OSAS) have an increased risk for being involved in motor-vehicle collisions. One of every five adults has at least mild Obstructive Sleep Apnea (OSA), and one of fifteen has at least moderate OSA. OSA has many negative effects, including excessive daytime sleepiness, increased risk of motor vehicle accidents, hypertension, psychological distress, and cognitive impairment. Apnea is generally considered the cessation of airflow for ten seconds or longer, and obstructive sleep apnea is apnea that occurs in spite of respiratory effort.
Alcohol and drugs can contribute to drowsiness. In one study in Italy, a significant number of drivers killed in single vehicle automobile accidents had consumed drugs. Cocaine was found to be the most frequently detected substance (9.5%), then benzodiazepines (7.5%), methadone, morphine and marijuana (THC) (3.5%). In 5.5% of subjects, more than one substance was found.
Use of drugs was found in another study to affect respiration rates. In all of the drugs tested as surrogates for morphine and heroine, inspiration was always depressed and there was a selective depression of expiration.
A large number of patents and technical papers address systems for detection of a drowsy driver but despite the potential for such systems, researchers have been largely unsuccessful in finding a feasible way to identify sleepiness or inattention. Nevertheless, it has been demonstrated that this can be done using an EEG. As one study reported “By combining power spectrum estimation, principal component analysis and artificial neural networks, we show that continuous, accurate, noninvasive, and near real-time estimation of an operator's global level of alertness is feasible using EEG measures.” (Estimating Alertness from the EEG Power Spectrum, January, 1997). However, attaching electrodes to a driver is intrusive and is not a practical solution to drowsiness detection. The disclosure below uses at least one electrical, physical, chemical and/or biological property of a driver, such as can be obtained by EEG- and EKG-signals, respiration, oxygen saturation, temperature, perspiration, etc., to reliably determine drowsiness in time to prevent accidents, see, for example, U.S. Pat. No. 7,215,991. This is accomplished unobtrusively without physically interacting and contacting with the driver, i.e., without attaching electrodes to the driver.
Another recently introduced product makes use of a cap which places electrodes on the skull or the operator and measures an approximate EEG and from the measurements attempts to determine whether the operator is fatigued. See WWW.smartcap.com.au.
Optical Applications
Many systems have been described in the patent and non-patent literature that attempt to use optical methods, such as video cameras, to detect blinking, head movement, yawning etc., as indications of the drowsy driver. The most successful study has led to the development of the PERCLOS method whereby the percentage of time that the eyes are closed is used to measure drowsiness. Unfortunately, none of these systems have demonstrated a forecasting success rate that permits action to be taken in time for the driver to get to a location where he/she can exit the road in time. Studies have shown that only ECG measurements have this ability to predict that a driver is falling asleep. At least one embodiment of the present invention, when using optical techniques for example, provides a method of measuring the heartbeat and respiration rates optically that, with appropriate analysis of the variability of these rates, will achieve a highly accurate prediction of the drowsiness state of the driver.
Previous studies have shown that eye closure occurrences dramatically increase during a 10-second period preceding an accident and propose that the monitoring such closures could allow the car to take some form of automated response to wake up the driver, e.g., a loud noise, a bright light, possibly even the activation of an “autopilot” if that capability is developed. Unfortunately, a 10-second warning is insufficient for the driver to find an exit and pull off of the road even if he/she accepts the warning of the vehicle system. Such systems must also be careful of a false alarm and thus must err on the side of not taking a drastic action.
It is also known that the duration of the eye closures one minute before an accident is much higher than at earlier times and partial eye closures (measured by the ratio between horizontal and vertical portions of the visible pupil) have been shown to be a way to detect drowsiness as much as about 10-12 minutes prior to an accident. However, measuring eye closures is hampered by a driver wearing glasses, hats or other apparatus which shades the eyes from the camera, intensity of ambient light (e.g., sunlight) and in variation in the eye shape from person to person. Temporary medical conditions, including use of medications, also can affect both the blink rate and the percentage of the eye that is closed. Thus, although there have been numerous studies and many systems developed, none has been deployed due to these and other factors. As one study reported, “The original aim of this project was to use the retinal reflection (only) as a means to finding the eyes on the face, and then using the absence of this reflection as a way of detecting when the eyes are closed. It was then found that this method might not be the best method of monitoring the eyes for two reasons. First, in lower lighting conditions, the amount of retinal reflection decreases; and second, if the person has small eyes, the reflection may not show.” Or as reported in another study, “Overall, the measures of driver drowsiness based on physical changes in the eye are developing into a technology that could potentially be used on the road. These approaches have a fundamental problem, however, in that the changes being measured are likely to be occurring late in the process of fatigue. It is possible that the driver has been through a significant period of high crash risk due to lowered alertness before significant eye closure effects are able to be detected. As a tool for fatigue prevention therefore, they will be signaling late stage fatigue and sleepiness when there are relatively few options for recovery apart from a period of sleep. Other problems associated with the eye- and face-change detection technologies, are deciding on the point at which the driver is in an unsafe state and when a warning should be applied, and deciding on the nature of the warning signal itself.” (Williamson, A., Chamberlain, T. “Review of on-road driver fatigue monitoring devices”, NSW Injury Risk Management Research Centre, University of New South Wales, April, 2005).
A commonly available device, the pulse oximeter, uses light to determine the pulse rate of a patient as well as the amount of oxygen in the blood. This is a device that usually clips to a finger, ear lobe or other part of the body, i.e., physically interacts and contacts the patient's body. It will be disclosed below that this can also be accomplished in an unobtrusive manner for the driver of a vehicle. As reported in U.S. Pat. No. 7,277,741, pulse oximetry takes advantage of the fact that in live human tissue, hemoglobin is a strong absorber of light between the wavelengths of 500 and 1100 nm. Pulsation of arterial blood through tissue is readily measurable, using light absorption by hemoglobin in this wavelength range. A graph of the arterial pulsation waveform as a function of time is referred to as an optical plethysmograph. The amplitude of the plethysmographic waveform varies as a function of the wavelength of the light used to measure it, as determined by the absorption properties of the blood pulsing through the arteries. By combining plethysmographic measurements at a plurality of different wavelength regions where oxy- and deoxy-hemoglobin have different absorption coefficients, e.g., two such wavelengths, the oxygen saturation of arterial blood can be estimated. Typical wavelengths employed in commercial pulse oximeters are 660 nm and 890 nm.
U.S. Pat. No. 7,822,453 further shows that an oximetry sensor can work on the forehead, although this is done by placing the sensor on the forehead. In addition, U.S. Pat. Appln. Publ. No. 20080045847 discloses that a non-contact, passive method for measurement of arterial pulse from areas such as the major superficial arteries of the body through analysis of thermal IR images acquired by passive thermal IR imaging sensors. The output waveform readily contains the information on heart rate, cardiac interbeat intervals, heart-rate variability and other features inherent in arterial pulse.
Other methods in the prior art are set forth in the following patent publications. U.S. Pat. No. 7,666,151 discloses use of piezoelectric film placed on a seat or bed surface that obtains a measure of both heartbeat and respiration rates. In this application, this concept is extended in what is considered an unobvious manner to use of a fluid-filled bladder weight sensor and load cell weight sensors as further methods of obtaining the heartbeat and respiration rates. Recently, The Plessey corporation has announced an EPIC ECG sensor (PS2501) which can be held in the hands of a patent and obtains the ECG signal. There is indication that this or a similar sensor can also do so if it is placed in the seatback in close proximity to the back of the driver. By whatever method is used, the analysis of these signals to obtain a measure of drowsiness is not believed to have previously been applied.
U.S. Pat. No. 6,822,573 discloses use of a geophone in the seat and also steering wheel-based sensors for heartbeat measurement, but does not disclose how to analyze this information to obtain a measure of drowsiness.
Assignee's U.S. Pat. Nos. 6,078,854, 6,253,134, 6,397,136, 6,330,501, 6,445,988, 6,474,683, 6,735,506, 6,736,231, 6,757,602, 6,950,022, 6,793,242, 7,050,897, 7,786,864 and 7,889,096 disclose general and radar methods for determining heartbeat rate as have others, such as a more recent U.S. Pat. No. 7,196,629 assigned to Bosch. Improvements to radar-based systems are also disclosed herein.
Analysis
Analysis of human heartbeat and respiration rates and in particular their variability during waking and sleeping has been reported in the literature. For example, a simple observation that the heartbeat rate shows less variability when sleeping but shows a noticeable jump on waking can be detected and used for drowsiness detection. As shown by several simulator tests, a driver often experiences a succession of micro-sleeps for several minutes prior to an accident. Thus, on a preliminary basis, the variability of the basic heartbeat rate shows a period of low variability followed by a jump on waking for each of these micro-sleeps and can, when applied in one embodiment of the invention, be a reliable predictive measure of drowsiness leading to an accident with a several minute lead time.
Spectral analysis of heart rate variability is well established as reported in (T. Penzel, J. W. Kantelhardt, H. F. Becker, J. H. Peter, A. Bunde, “Detrended Fluctuation Analysis and Spectral Analysis of Heart Rate Variability for Sleep”, Hospital of Philipps-University, Marburg, Germany, Nov. 16, 2003): “The physiological interpretation of the very low-frequency (VLF) component (<0.04 Hz) is still discussed, the low-frequency (LF) component (0.04-0.15 Hz) reflects baroreflex sympathetic control of blood pressure, and the high-frequency (HF) component (0.15-0.4 Hz) reflects respiratory rhythm and is believed to be related to parasympathetic control of heart rate.”
From Wikipedia: “Sympathetic and parasympathetic divisions typically function in opposition to each other. This natural opposition is better understood as complementary in nature rather than antagonistic. For an analogy, one may think of the sympathetic division as the accelerator and the parasympathetic division as the brake. The sympathetic division typically functions in actions requiring quick responses. The parasympathetic division functions with actions that do not require immediate reaction. A useful acronym to summarize the functions of the parasympathetic nervous system is SLUDD (salivation, lacrimation, urination, digestion and defecation).”
A discussion of respiration variability can be found in U.S. Pat. No. 7,397,382. “ . . . a depth of breathing of a person is detected, and drowsiness of the person is determined when the depth of breathing falls in a predetermined breathing condition including at least one of a sudden decrease in the depth of breathing and a periodic repetition of deep breathing and shallow breathing.” “ . . . the thorax pressure generally changes in a fixed manner when a person condition changes from awakened to drowsing. When the person starts to feel drowsy from the awakened condition or from the rest condition, the depth and period of breathing does not remain stable and the breathing sometimes cannot be found. In this instance, the depth of the breathing suddenly becomes shallow or alternately becomes deep and shallow in three to seven breaths. When the person falls asleep, the breathing is repeated periodically although the depth of breathing slightly changes.” This patent uses a steering wheel-mounted sensor to detect the driver's pulse and then derives a measure of respiration from the pulse signal. This is dependent on the driver gripping the steering wheel where the sensor is located with sufficient force as to allow accurate measurements to be made. This condition is generally not reliably achieved in practice due to the many forms of driver steering wheel interaction, especially when drowsiness in occurring. Such a system, for example, will not work if the driver is wearing thick gloves.
Countermeasures
Various studies have evaluated the effects of various countermeasures. One study evaluated a fatigue warning system that used ocular and face monitoring, vehicle speed, steering position and lane position. They found that in spite of frequent warnings, users of the system did not take more or longer breaks. Drivers generally ignored the warning signals received. The physical aspect of the warning signals used in the study had no impact on driver fatigue levels. Voluntary rest stops, lasting on average about thirty minutes, only had a minor impact on decreasing driver fatigue with short-lived effects. The authors concluded that voluntary breaks were ineffective in substantially counteracting the effects of fatigue associated with prolonged night-time driving.
In contrast, Mercedes Benz has developed a technology reminder on the series E-Class cars in the year 2010. This technology is able to monitor seventy different parameters to detect fatigue without reporting on what these parameters were. If drowsiness is detected, a coffee cup icon and the words “time to rest” appears on the dashboard panel accompanied by the sound of car alarms to remind the driver so as not to fall asleep while driving. Reduction of accidents due to this system is recorded to be one-third of the total rate.
Another reported technique for detecting drowsiness is by monitoring a response of the driver. This involves periodically requesting the driver to send a response to the system to indicate alertness. A problem with this technique is that it will eventually become tiresome and annoying to the driver. This reported technique thus teaches away from this concept which is used in some embodiments of the invention. This very useful technique must be applied in such a manner that it does not become tiresome to the driver.
Until now, use of heartbeat and respiration variability monitoring through electric field, optical, Plessey ECG sensors or weight monitoring systems to detect drowsiness in vehicle occupants has not previously been disclosed, to the extent this field has been investigated by the inventor. Additional background of the invention is found in the related applications. All of the patents, patent applications, technical papers and other references mentioned herein are incorporated by reference in their entirety.
Technical papers and other published documents that are particularly relevant to the inventions described herein include:    1. Borgobello, Bridget “MIT developing webcam-based health monitoring minor”. Press release MIT News Office, Oct. 5, 2010.    2. J. Smith, T. White, C. Dodge, J. Paradiso, N. Gershenfeld, D. Allport “Electric Field Sensing for Graphical Interfaces”. 1998, IEEE Comput. Graph. Appl.    3. “Location Privacy And Wireless Body Area Networks”, The Physics asXiv Blog, MIT Technology Review, Mar. 23, 2011.    4. Richards, Austin, Alien Vision: Exploring the Electromagnetic Spectrum with Imaging Technology (SPIE Press Monograph Vol. PM104), 2001, ISBN 0-8194-4142-2    5. Poh, et al. “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation”, Optics Express, Vol. 18, Issue 10, pp. 10762-10774 (2010).    6. W. J. Cui, L. E. Ostrander, and B. Y. Lee, “In vivo reflectance of blood and tissue as a function of light wavelength,” IEEE Trans. Biomed. Eng. 37(6), 632-639 (1990).    7. Grubb et at, “Towards Forehead Reflectance Photoplethysmography to Aid Delivery Room Resuscitation in Newborns”, 4th European Conference of the International Federation for or Medical and Biological Engineering 23-27 Nov. 2008, Antwerp, Belgium (IFMBE Proceedings).    8. “Location Privacy And Wireless Body Area Networks” The Physics arXiv Blog, MIT Technology Review, 03/23/11.    9. W. W. Wierwille “Overview of Research on Driver Drowsiness Definition and Driver Drowsiness Detection”, Paper No. 94 S3 0 07, Virginia Polytechnic Institute and State University May 23, 1994.    10. J. A. Paradiso, N. Gershenfeld, “Musical Applications of Electric Field Sensing”, Computer Music Journal 1997.    11. Tzyy-Ping Jung, Scott Makeig, Magnus Stensmo, Terrence J. Sejnowski, “Estimating Alertness from the EEG Power Spectrum”, IEEE Transactions On Biomedical Engineering, VOL. 4, NO. I, J MAR) 1997.    12. D. W. Rowe, J. Sibert, D. Irwin, “Heart Rate Variability Indicator of User State as an Aid to Human-Computer Interaction”, CHI '98 Proceedings of the SIGCHI conference on Human factors in computing systems, 1998.    13. James Nolan, MD; Phillip D. Batin, MD; Richard Andrews, MRCP; Steven J. Lindsay, MRCP; Paul Brooksby, MD; Michael Mullen, MRCP; Wazir Baig, MD; Andrew D. Flapan, MD; Alan Cowley, FRCP; Robin J. Prescott, PhD; James M. M. Neilson, PhD; Keith A. A. Fox, FRCP, “Prospective Study of Heart Rate Variability and Mortality in Chronic Heart Failure”, Circulation. 1998; 98:1510-1516, doi: 10.1161/01.CIR.98.15.1510, (Circulation. 1998; 98:1510-1516.), 1998 American Heart Association, Inc.    14. “Drowsiness Detector (Stanford MS EE Project)—MIT Media Lab.”, May 30, 2001.    15. “Eye-Activity Measures of Fatigue and Napping as a Fatigue Countermeasure”, Final Report, USDOT, FHWA-MC-99-028, January, 1999.    16. N. Parmar “Drowsy Driver Detection System”, Design project, Department of Electrical and Computer Engineering, Ryerson University. 2002.    17. Smith, P.; Shah, M.; da Vitoria Lobo, N., “Monitoring Head Eye Motion for Driver Alertness with One Camera”, Pattern Recognition, 2000. Proceedings. 15th International Conference on, Issue Date: 2000, On page(s): 636-642, vol. 4.    18. K. Willson, D. P. Francis, R. Wensel, A. J. S. Coats, and K. H. Parker, “Detrended Fluctuation Analysis and Spectral Analysis of Heart Rate Variability for Sleep”, 2002 Physiol. Meas. 23 385 doi:10.1088/0967-3334/23/2/314.
19. S. Yu. Chekmenev, H. Rara, and Aly A. Farag, “Non-contact, Wavelet-based Measurement of Vital Signs using Thermal Imaging”, ICGST Int J Graph Vision Image Process 2006; 6:25-30. 35.    20. Amy Diane Droitcour, “Non-contact measurement of heart and respiration Rates with a single-chip microwave doppler radar”, Stanford University Doctoral Dissertation, June, 2006.    21. W. Jiang, Z. Chongxunr, L. Guohua and D. Ming, “A New Method for Identifying the Life Parameters via Radar”, URASIP Journal on Applied Signal Processing, Volume 2007 Issue 1, 1 Jan. 2007.    22. E. P. Scilingo, A. Lanat, D. Zito, D. Pepe, “Wearable monitoring of cardiopulmonary activity through radiant sensing”. Proceedings of the phealth2008.    23. J. F. Layerle, X. Savatier, J. Y. Ertaud, “Catadioptric Sensor for a Simultaneous Tracking of the Driver's Face and the Road Scene” The 8th Workshop on Omnidirectional Vision, Camera Networks and Non-classical Cameras—OMNIVIS, Marseille: France (2008).    24. G. D. Furman, A. Baharav, C. Cahan, “Early Detection of Falling Asleep at the wheel” Cardiology, 2008, 2008—ieeexplore.ieee.org.    25. Bridget Borgobello, “MIT developing webcam-based health monitoring minor”, MIT News Office, Oct. 5, 2010.    26. Furman, G. D.; Baharav, A, “Investigation of Drowsiness while Driving Utilizing Analysis of Heart Rate Fluctuations”, IEEE Conference on Computing in Cardiology, 2010, Issue Date: 26-29 Sep. 2010, page(s): 1091-1094.27. Kate Greene, “Talking to the Wall”, MIT Technology Review May 3, 2011.
U.S. patents that are particularly relative to the inventions described herein include the following in addition to those referenced in the text:    27. U.S. Pat. No. 6,684,973, entitled “Occupant detecting apparatus”    28. U.S. Pat. No. 6,816,077, entitled “Multiple sensor vehicle occupant detection”    29. U.S. Pat. No. 6,960,841, entitled “Passenger detection system and detection method”
Possible definitions of terms used in the application are set forth in U.S. patent application Ser. No. 10/940,881, incorporated by reference herein.